Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding

Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that is less explored in as it often redundant to textual information. In this paper, we propose an Object-level Context modeling framework (OVC) efficiently capture explore translation. With detected objects, the proposed OVC encourages ground desirable objects by masking irrelevant modality. We equip with additional object-masking loss achieve goal. The estimated according similarity between masked source texts so encourage source-irrelevant objects. Additionally, order generate vision-consistent target words, further a vision-weighted OVC. Experiments datasets demonstrate model outperforms state-of-the-art analyses show helps MMT.